reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerJeremy Bícha <jbicha@ubuntu.com>
Fri, 22 Dec 2023 15:32:30 +0000 (10:32 -0500)
commitbd784c38c5bc96156581da0109bfb3db263db8e6
tree27ec52a3b8804e1f7e08945b6049fd6daf441b80
parentffd451536acf909582c6be0acbedbef0155f3fc5
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, with additional inspiration
from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/reftests/gtk-reftest.c